Add agentic review for management AutoPRs - #50039
Open
Weidong Xu (weidongxu-microsoft) wants to merge 14 commits into
Open
Add agentic review for management AutoPRs#50039Weidong Xu (weidongxu-microsoft) wants to merge 14 commits into
Weidong Xu (weidongxu-microsoft) wants to merge 14 commits into
Conversation
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8cab2a2a-1ca4-44ea-8836-2c9b6dc2cfa9
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8cab2a2a-1ca4-44ea-8836-2c9b6dc2cfa9
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8cab2a2a-1ca4-44ea-8836-2c9b6dc2cfa9
Embed the critic contract in the inline sub-agent and runtime-import trusted reviewer rules into the main prompt so branch trials do not depend on files already existing on Azure main. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8cab2a2a-1ca4-44ea-8836-2c9b6dc2cfa9
Use GA CHANGELOG breaking entries as the primary evidence, ignore beta breaks and generated paths, and derive folder identity from the third module-name segment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8cab2a2a-1ca4-44ea-8836-2c9b6dc2cfa9
Treat a missing release-plan link as a nonblocking PR-description concern so GA breaking changes remain visible. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8cab2a2a-1ca4-44ea-8836-2c9b6dc2cfa9
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Azure Pipelines: Successfully started running 1 pipeline(s). 34 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Member
Author
|
We've got a few month's review on my local agent, and we've collected some typical pattern to raise. Now add a agentic workflow. We will fix/enhance it as we see more cases. |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Member
Author
|
Validation completed on the current branch:
|
Weidong Xu (weidongxu-microsoft)
marked this pull request as ready for review
August 5, 2026 15:55
Weidong Xu (weidongxu-microsoft)
requested a review
from a team
as a code owner
August 5, 2026 15:55
|
Azure Pipelines: Successfully started running 1 pipeline(s). 34 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Copilot started reviewing on behalf of
Weidong Xu (weidongxu-microsoft)
August 5, 2026 15:55
View session
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a GitHub Agentic Workflows-based, read-only “management AutoPR” reviewer for Java management-library generated PRs, including a dedicated reviewer persona, a false-positive “critic” sub-agent protocol, and a Vally evaluation suite to validate findings/orchestration/true-negatives behavior.
Changes:
- Introduces a
pull_request_targetworkflow (plus compiled lock) that gates on eligible AutoPRs and posts at most one advisory comment via safe-outputs. - Adds management-specific review rules (folder/version/LRO/api-version overlap/breaking/release-plan) and a critic protocol for false-positive filtering.
- Adds a Vally eval suite with fixtures to validate detection, orchestration, and true-negative behavior; updates cspell dictionary for new terms.
Reviewed changes
Copilot reviewed 27 out of 28 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| .vscode/cspell.json | Adds AutoPR/eval fixture vocabulary to avoid spellcheck noise. |
| .github/workflows/management-autopr-review.md | Defines the management AutoPR review workflow prompt, gates, safe-outputs, and critic dispatch contract. |
| .github/workflows/management-autopr-review.lock.yml | Compiled gh-aw workflow used by GitHub Actions to run the reviewer. |
| .github/skills/management-autopr-review/SKILL.md | Management-specific review rules and output contract for the unattended reviewer. |
| .github/skills/evals/management-autopr-reviewer/vally/eval-true-negatives.yaml | True-negative suite ensuring low-noise behavior and deduplication. |
| .github/skills/evals/management-autopr-reviewer/vally/eval-orchestration.yaml | Orchestration tests for injection resistance and critic usage. |
| .github/skills/evals/management-autopr-reviewer/vally/eval-findings.yaml | Positive detection coverage for each supported concern type. |
| .github/skills/evals/management-autopr-reviewer/run-evals.ps1 | Helper script to run Vally suites locally and emit junit output. |
| .github/skills/evals/management-autopr-reviewer/README.md | Documents the eval intent, constraints, and how to run suites. |
| .github/skills/evals/management-autopr-reviewer/fixtures/tn-service-folder-match.md | Fixture for service-folder match true-negative. |
| .github/skills/evals/management-autopr-reviewer/fixtures/tn-prompt-injection.md | Fixture for prompt-injection resistance true-negative. |
| .github/skills/evals/management-autopr-reviewer/fixtures/tn-prior-concern.md | Fixture for carried-forward concern deduplication behavior. |
| .github/skills/evals/management-autopr-reviewer/fixtures/tn-generated-only-java.md | Fixture ensuring generated-path-only Java changes are ignored. |
| .github/skills/evals/management-autopr-reviewer/fixtures/tn-beta-breaking-changelog.md | Fixture ensuring MGMT-BREAKING is not raised for beta packages. |
| .github/skills/evals/management-autopr-reviewer/fixtures/tn-additive-beta-update.md | Fixture for additive beta update true-negative (no concerns). |
| .github/skills/evals/management-autopr-reviewer/fixtures/positive-version.md | Fixture for MGMT-VERSION detection (stable package on preview API). |
| .github/skills/evals/management-autopr-reviewer/fixtures/positive-service-folder-mismatch.md | Fixture for MGMT-FOLDER detection. |
| .github/skills/evals/management-autopr-reviewer/fixtures/positive-missing-release-plan.md | Fixture for MGMT-RELEASE-PLAN detection. |
| .github/skills/evals/management-autopr-reviewer/fixtures/positive-lro.md | Fixture for MGMT-LRO detection. |
| .github/skills/evals/management-autopr-reviewer/fixtures/positive-folder.md | Fixture for MGMT-FOLDER detection in a “collision” scenario. |
| .github/skills/evals/management-autopr-reviewer/fixtures/positive-breaking.md | Fixture for MGMT-BREAKING detection based on GA changelog section. |
| .github/skills/evals/management-autopr-reviewer/fixtures/positive-api-version-overlap.md | Fixture for MGMT-API-VERSION-OVERLAP detection. |
| .github/skills/evals/management-autopr-reviewer/.vally.yaml | Defines Vally suites (all, true-negatives) and paths. |
| .github/skills/evals/management-autopr-reviewer/.gitignore | Ignores eval results output directory. |
| .github/aw/actions-lock.json | Updates pinned gh-aw setup action version used by compilation. |
| .github/agents/protocols/management-autopr-review-critic.protocol.md | Defines the reviewer→critic dispatch contract and verdict semantics. |
| .github/agents/management-autopr-reviewer.agent.md | Reviewer persona, calibration, gates, and critic-dispatch sequence. |
| .github/agents/management-autopr-review-critic.agent.md | Critic persona instructions for filtering false positives. |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Member
Author
|
Real-PR dry runs were completed from the private trial host. Safe outputs were staged, so no Azure PR was modified.
|
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation
vally lint .github/skills/management-autopr-review